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1.  Introduction 


The  U.S.  Defense  Advanced  Research  Project  Agency  (DARPA)  and  the  U.S.  Joint  Forces 
Command  (JFCOM)  developed  a  suite  of  decision  support  tools  to  enhance  the  capability  of 
commanders  and  staffs  to  plan  and  execute  effects-based  campaigns.  The  complex  environments 
in  which  commanders  and  staffs  must  operate  present  such  nonmilitary  issues  as  political, 
economic,  social,  information,  and  infrastructure  (PMESII).  To  better  assess,  understand,  and 
explore  the  relevant  and  important  issues  across  these  operational  dimensions,  DARPA  and 
JFCOM  have  developed  the  Conflict  Modeling,  Planning,  and  Outcomes  Experimentation 
Program  (COMPOEX)  system  to  support  the  commanders  and  staffs.  Multiple  limited  objective 
experiments  (EOEs)  were  conducted  to  test  and  assess  the  progress  and  effectiveness  of  one  or 
more  of  the  tools.  This  report  describes  the  EOEs  addressing  the  effects  on  command  and  control 
(C2)  and  planning  for  the  Option  Exploration  Tool  (OET)  and  the  Campaign  Planning  Tool 
(CPT).  The  COMPOEX  EOE  2-3  was  designed  to  test  the  use  of  multi-resolution  models 
(MRMs)  in  conjunction  with  the  OET  and  CPT  for  operational  level  planning  using  the  effects- 
based  operations  approach.  The  OET  and  the  CPT  are  the  primary  COMPOEX  interfaces  with 
the  MRMs.  The  CPT  provides  a  timeline  to  designate  the  actions,  effects,  resources,  and 
objectives  for  the  plan  (figure  1).  These  plan  elements  provide  input  into  the  MRMs  and 
designate  which  aspects  of  the  plan  to  be  simulated.  The  results  of  the  simulation  can  then  be 
examined  using  the  OET. 
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Figure  1 .  Example  of  the  CPT. 


The  OET  consists  of  several  “views”  of  the  simulated  plan  that  represent  the  results  of  the 
simulated  run  given  the  plan  inputs  provided  through  the  CPT.  The  views  include  the 
(1)  outcome  view,  (2)  graph  view,  (3)  panel  view,  (4)  exploration  analysis  view,  and  (5)  model 
operations  view  (appendix  A).  The  outcome  view  presents  the  events  or  actions  that  exceed  a 
predefined  threshold  while  the  graph  view  visually  presents  the  simulated  results  across  time  for 
the  selected  variables.  The  panel  and  model  operations  views  show  the  relationships  between  the 
variables  while  the  exploration  analysis  view  illustrates  the  variables  influencing  and  responsible 
for  the  change  in  the  targeted  variable  as  well  as  what  variables  it  in  turn  affects. 
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The  planning  environment  that  was  used  for  this  event  was  the  Afghanistan  region,  with  the 
objectives  of  developing  multiple  lines  of  effort  (e.g.,  security,  governance,  rule  of  law)  and 
supporting  effects  to  improve  regional  stability  of  the  Afghanistan  region.  A  line  of  effort  is  a 
PMESITrelated  area  of  effort  that  needs  to  be  planned  and  coordinated  to  achieve  the  objectives. 
The  basic  concept  of  operation  (CONOP)  established  for  using  COMPOEX  is  illustrated  in  the 
following  six  steps  (COMPOEX,  2007),  with  an  emphasis  on  steps  4-6  for  this  event. 

1.  Mission  Analysis  -  Decomposition  of  the  commander’s  intent  into  key  elements,  dates, 
constraints  and  restraints,  etc.  Identify  the  actors,  issues  and  sources  of  conflict;  develop  a 
theory  of  the  dynamics  between  actors  and  the  drivers  of  conflict,  competition,  and 
cooperation  among  actors. 

2.  Situation  Assessment  -  Review  of  the  MRM  and  baseline  estimates  of  projected  PMESII 
dynamics  without  any  planned  exogenous  actions. 

3.  Strategy  Development  -  Identification  of  objectives  on  a  timeline.  Articulate  specific 
desired  ends,  objective,  and  purpose  of  the  mission. 

4.  Plan  Development  -  Insertion  of  specific  planned  diplomatic,  information,  military, 
economic  (DIME)  actions  along  lines  of  effort. 

5.  Run  Simulation  -  Conduct  a  run  of  the  simulation  to  produce  PMESII  effects  of  the 
scheduled  DIME  actions  across  all  model  components.  Apply  the  actions  identified  in  the 
courses  of  action  (COAs)  to  system  models  that  synthesize  the  expected  effects  from 
combined  actions.  Observe  first-order  and  higher  order  effects;  identify  critical  interactions 
and  potential  unintended  consequences.  Test  and  validate  plan  and/or  plan  fragments  by 
execution  of  the  plan  within  the  full  PMESII  model  simulation. 

6.  View  Effects  -  The  user  views  the  effects  of  the  candidate  campaign  on  the  targeted 
systems.  Apply  these  observations  of  the  actual  outcomes  of  individual  actions,  and 
aggregate  outcomes  to  update  and  refine  the  models  used  in  analysis  and  wargaming. 

Eigure  2  shows  the  six  steps  of  a  CONOP  for  COMPOEX. 


2 


The  participants  were  assigned  to  three  primary  planning  groups,  each  of  which  was  responsible 
for  the  following  lines  of  effort:  (1)  reconstruction,  (2)  governance,  and  (3)  security.  The 
assignment  of  participants  to  each  of  these  lines  of  effort  was  based  roughly  on  the  participants’ 
past  experience  and  background.  Each  of  these  lines-of-effort  planning  teams  had  a  military  or 
civilian  lead  experienced  in  that  particular  area.  In  addition  to  the  three  lines-of-effort  teams, 
there  was  a  strategy  cell  that  convened  periodically  throughout  the  event  to  discuss  strategic  and 
overarching  issues. 

The  enhanced  ability  of  COMPOEX  to  simulate  a  plan  and  the  various  components  within  a  plan 
(e.g.,  actions,  effects)  is  expected  to  positively  stimulate  the  thought  processes  of  the  commander 
and  planning  staff.  The  exploration  of  various  “what-if  ’  scenarios  and  plans  through  simulation 
allows  commanders  to  further  develop  their  understanding  and  test  various  theories  of  conflict 
and  effects  of  key  perceived  relationships  across  PMESII  elements.  This  increased  capacity  to 
explore  the  environment  and  situation  through  the  MRMs  and  examination  of  the  results  through 
the  OET  is  hypothesized  to  improve  planning  effectiveness  and  understanding.  It  is  also 
hypothesized  that  MRMs  of  sufficient  validity  and  fidelity  can  be  developed  to  facilitate 
planning  with  limited  time  and  resources. 
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2.  Method 


2.1  Participants 

There  were  25  active  participants  for  LOE  2-3,  consisting  of  eight  senior  mentors  and  17 
planners.  Fifteen  of  the  players  had  military  background  and  experience  while  10  of  the  players 
had  a  nonmilitary  (government  or  civilian)  background  and  experience.  The  total  number  of 
years  of  military  experience  (412)  slightly  outweighed  civilian  experience  (336.5).  Overall 
tactical  planning  experience  (247)  was  most  prevalent,  with  significant  amounts  evident  for  the 
operational  (201)  and  strategic  (130)  levels.  Experience  in  the  Afghanistan  environment  was 
limited — a  total  of  31.5  years  of  experience  across  all  participants.  This  experience  was  spread 
among  13  participants,  with  a  maximum  of  7  years  of  experience  for  one  participant. 

2.2  Materials 

All  participants  received  the  read-aheads  on  the  Afghanistan  region  and  the  Theory  of  Conflict 
(ToC)  work  that  had  been  completed  prior  to  EOF  2-3.  Participants  had  access  to  the 
COMPOEX  system  using  laptop  computers  during  the  EOF.  Multiple  overhead  digital  projectors 
were  available  to  display  images  on  screens  in  the  EOF  area. 

2.3  Procedure 

EOF  2-3  was  a  2- week  event  with  training  and  execution  components.  The  first  week  was 
dedicated  to  training  participants  on  the  COMPOEX  system,  and  the  second  consisted  of 
execution  set  week  in  the  Afghanistan  region  using  COMPOEX.  The  execution  week  began  with 
the  development  of  10  effects  by  the  three  line-of-effort  teams  (reconstruction,  governance,  and 
security.  During  the  execution  week,  the  line-of-effort  planning  teams  developed  the  individual 
effects  into  a  single  plan  for  their  line  of  effort.  The  teams  continued  refining  the  plan  by 
integrating  the  three  lines-of-effort  plans  into  a  single  integrated  plan  by  the  end  of  the  event. 
Resources  were  allocated  to  additional  effects  and  lines  of  effort  as  needed  throughout  the 
execution  week.  The  strategy  cell  and  the  leaders  of  each  line  of  effort  made  the  determination 
for  these  additional  effects  and  lines  of  effort. 
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3.  Results  and  Discussion 


3.1  Plan  Output 

3.1.1  MRM  and  Theory  of  Conflict  Development 

The  development  of  the  MRMs  was  done  prior  to  the  LOE  2-3  and  amounted  to  approximately 
600  man-hours  over  a  3.5-month  period.  This  effort  consisted  of  10  individuals  working  part- 
and  full-time.  The  comparative  planning  cell  also  had  10  individuals  developing  their  nodal 
network  (network  of  relevant  people,  places,  and  things)  and  database.  The  total  time  spent 
developing  it  when  LOE  2-3  occurred  was  5  months  or  approximately  8000  man-hours. 

In  conjunction  with  the  time  spent  developing  the  MRMs,  a  ToC  workshop  was  conducted  in 
May  2007.  A  ToC  identifies  and  defines  the  relevant  elements  and  structure  of  a  PMESII  system 
along  with  the  causal  explanation  of  the  system’s  basic  behaviors  {COMPOEX,  2007).  This 
2-day  ToC  workshop  collected  expertise  on  the  Afghanistan  region  from  military  and  civilian 
government  and  nongovernment  agencies  to  develop  multiple  possible  Afghanistan-related 
ToCs.  Approximately  145  man-hours  of  Afghanistan  expertise  was  applied  to  the  development 
of  ToCs  and  was  included  into  the  building  of  the  MRMs.  The  total  amount  of  man-hours 
including  the  ToC  workshop  was  745.  The  amount  and  access  to  Afghanistan  subject  matter 
experts  (SMEs)  by  the  comparative  planning  cell  is  unknown. 

3.1.2  Plan  Products 

The  participants  comprising  the  command  and  staff  groups  developed  a  number  of  products  as 
they  engaged  in  the  execution  week  with  the  COMPOEX  system.  Actions  in  a  plan  are  one  of  the 
primary  planning  components  that  the  COMPOEX  system  allows  the  planner  to  explore.  This  is 
done  by  defining  the  action(s)  within  the  CPT,  and  is  simulated  against  the  MRMs  to  allow  the 
commander  and/or  staff  to  examine  the  effects  of  their  actions.  A  total  of  592  unique  actions 
were  defined  and  explored  by  the  command  and  staff  groups  for  this  event  over  the  5-day  event 
period. 

The  material  and  domain  used  during  the  execution  week  was  also  used  during  part  of  the 
training  week.  The  inclusion  of  the  actions  created  and  examined  during  the  training  period,  an 
additional  71  actions,  totaled  663  actions  during  the  training  and  execution  weeks.  As  a  general 
reference  point,  the  Standing  Joint  Eorce  Headquarters  “starter  kit”  of  actions  used  by  the 
comparative  planning  cell  consisted  of  150  actions,  which  have  only  been  slightly  modified. 

Ten  effects  were  developed  and  integrated  into  the  final  plan  during  the  training  and  execution 
weeks.  Six  additional  effects  were  initiated  across  the  three  lines  of  effort,  and  one  additional 
line  of  effort  was  created  as  the  issue  of  strategic  communications  appeared  to  have  a  prominent 
effect  across  each  of  the  existing  lines  of  effort.  The  original  10  effects  were  developed  in  the 
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span  of  the  5-day  event  and  integrated  into  a  single  plan  across  four  lines  of  effort  that  included 
the  reconstruction,  governance,  security,  and  strategic  communications  lines  of  effort,  while  the 
additional  six  effects  were  being  developed  concurrently  beginning  on  day  8.  The  comparative 
planning  cell  had  developed  five  effects  over  a  period  longer  than  2  months  at  the  time  LOE  2-3 
occurred.  The  efficiency  of  the  COMPOEX  team  can  be  described  by  considering  the  number  of 
effects  (16)  initiated  during  the  number  of  man-hours  used  for  their  development.  The  total  man¬ 
hours  during  the  execution  week  was  1000  (40  h  x  25  participants),  which  is  62.5  man-hours  per 
effect. 

3.2  Coordination 

The  plan  output  was  produced  by  the  participants  during  the  execution  event.  This  section  will 
focus  on  the  processes  and  interaction  patterns  that  were  employed.  Traditional  statistical 
analysis  complemented  by  social  network  analysis  was  used  to  analyze  the  patterns  of 
interaction. 

Social  network  analysis  is  based  on  network  theory,  which  uses  graphs  as  a  representation  of 
symmetric  or  asymmetric  relations  between  discrete  objects  (Scott,  2000).  The  graph  is  a 
mathematical  structure  used  to  represent  pair-wise  relations  between  objects.  Placed  within  a 
social  context  of  humans  and  their  interactions,  a  social  network  is  a  set  of  individuals  (i.e., 
nodes)  connected  through  social  interactions  like  face-to-face  or  email  communication  (i.e., 
links).  The  analysis  of  these  social  networks  consists  of  a  family  of  relational  methods  to 
systematically  uncover  patterns  of  interconnectedness. 

The  general  patterns  of  interaction  will  be  described  for  the  governance  line-of-effort  team,  as  it 
currently  is  the  most  complete  data  set  among  the  lines  of  effort  and  included  the  commander  as 
its  lead.  Eigure  3  displays  the  social  network  graphs  for  the  first  4  days  of  the  execution  week. 
The  last  day  was  excluded  from  this  analysis  because  the  participants  were  primarily  working  on 
preparations  and  execution  of  the  final  brief  and  the  event  review.  The  nodes  in  the  network 
graphs  are  color  coded  to  represent  members  of  each  of  the  lines  of  effort.  Days  1  and  2 
represent  few  interactions  outside  the  governance  team,  as  indicated  by  only  one  link  to  the 
security  team  and  four  links  with  the  reconstruction  team  during  day  1 .  Day  2  followed  a  similar 
pattern  with  only  three  links  with  the  reconstructions  team.  Eor  both  days,  P22  is  a  critical  bridge 
between  the  governance  and  reconstruction  teams.  Day  3  represents  a  day  of  primarily  internal 
work  by  the  governance  group  with  only  one  link  to  the  security  team.  The  density  of  the 
network  also  suggests  that  communication  was  focused  between  a  few  individuals  that  formed  a 
communication  triad  (P15,  P16,  and  P17),  comprising  92%  of  the  communication  by  the 
governance  team  that  day.  Day  3  appears  to  be  the  quiet  before  the  storm,  as  Day  4  shows  a 
flurry  of  communication  activity  both  internal  and  external  to  the  governance  team.  As  figure  3D 
shows,  there  are  multiple  links  to  both  the  security  and  reconstruction  teams,  as  day  4 
represented  the  primary  merging  effort  across  the  lines  of  effort. 
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Figure  3.  Patterns  of  communication  for  the  governance  line-of-effort  team  across  the  first  4  days  of  the  execution  week.  Governance  members  are  teal, 
reconstruction  is  red,  security  is  blue,  and  yellow-green  is  strategic  cell  and  “other.” 


Further  examination  of  the  patterns  of  interaction  collected  through  observational  techniques 
within  the  governance  line  of  effort  also  provided  information  on  the  reliance  of  the  governance 
team  on  the  COMPOEX  tool  and  model  developers.  All  interactions  with  the  COMPOEX  tool 
and  model  developers  that  attended  EOE  2-3  in  a  support  role  were  coded  together  and 
represented  as  the  P27  node  in  figure  3A-D.  Day  1  shows  a  strongly  central  role  and  reliance  on 
the  tool  support  staff,  as  P27  is  a  central  node  in  the  network  with  many  communication  ties  to 
other  governance  team  members.  As  time  progresses  from  day  1  through  4,  the  position  of  the 
support  staff  (P27)  gradually  drifts  to  the  periphery  of  the  communication  network,  which 
represents  a  declining  reliance  on  the  support  staff  for  the  team  to  efficiently  function  with  the 
COMPOEX  system.  This  trend  in  the  communication  interactions  was  supported  by  the 
reduction  of  total  time  spent  in  communication  with  the  COMPOEX  tool  and  model  support  staff 
(figure  4).  Both  of  these  findings  are  important  because  of  the  need  of  the  COMPOEX  tool  to  be 
independent  of  a  large  support  staff  and  easy  to  use  as  it  is  transitioned  to  field  settings. 

COMPOEX  Tool  Support  Interactions 
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Figure  4.  Total  time  spent  in  minutes  by  the  governance  team  interacting  with  the 
COMPOEX  tool  and  model  support  staff  over  time. 


Detailed  interactions  were  recorded  for  each  of  the  three  members  of  the  command  group,  which 
consisted  of  the  commander  and  his  two  deputies.  Each  member  of  the  command  group  directed 
the  actions  of  the  three  primary  lines  of  effort  (reconstruction,  governance,  and  security).  The 
interactions  recorded  included  those  with  members  of  the  planning  teams  (internal  and  external 
to  their  line  of  effort)  and  those  with  the  specific  COMPOEX  tools  used.  The  dynamic 
interactions  by  the  three  members  of  the  command  group  were  examined  and  analyzed  using  the 
Social  Network  Image  Animator  (SoNIA).  The  network  illustration  showed  interactions  by  the 
command  group  with  the  tools  decreased  as  interactions  with  the  planning  team  members 
increased  (video  available  upon  request). 
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The  negative  trend  in  COMPOEX  tool  usage  by  the  eommand  group  is  eonfirmed  through  an 
analysis  of  overall  time  spent  across  all  the  tools  over  time  (figure  5).  As  evident  in  the  SoNIA 
analysis,  there  are  individual  differences  evident  among  tool  usage  but  the  decrease  in  tool  usage 
from  each  of  the  command  group  members  shows  their  transition  from  hands-on  tool  use  to  a 
more  hands-off  command  orientation.  This  pattern  of  interactions  represents  a  C2  shift  from  a 
focus  on  the  tool  to  a  focus  on  the  planner  who  is  using  the  COMPOEX  tools  to  support  their 
plans  and  conclusions. 


COMPOEX  Tool  Use  Among  Leadership  -  Total  Time 


450 


Figure  5.  Total  time  spent  by  each  command  group  member  with  the  COMPOEX  tools  over  time. 


3.3  COMPOEX  Tool  Effects  on  Human  Performance 

The  two  previous  sections  presented  the  results  of  what  the  participants  produced  during  the 
event  (plan  output)  and  how  they  produced  those  products  (coordination).  This  section  will 
present  the  results  of  both  on  human  performance  effects  regarding  understanding,  workload,  and 
trust  in  the  MRMs. 

Gilbert  and  Troitzsch  (2005)  succinctly  and  accurately  summarize  the  primary  purpose  of 
modeling  and  simulation:  “Simulation  is  the  development  of  explanations  rather  than  the 
prediction  of  specific  outcomes”  (p.  12).  This  is  the  primary  purpose  of  the  MRMs  within 
COMPOEX,  as  the  MRMs  attempt  to  facilitate  the  development  of  explanations  and  help 
enhance  understanding  of  the  operational  environment.  The  COMPOEX  MRMs  and  models  or 
simulations  in  general  can  improve  the  explorability  of  the  target  domain  by  creating  the  ability 
to  experiment  with  different  options  related  to  the  domain.  It  is  through  this  experimentation 
process  that  humans  learn  about  their  environment  beginning  from  childhood  (Siegler,  2005)  and 
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continuing  through  adulthood  (Norman,  1988).  This  section  will  investigate  whether  or  not  the 
COMPOEX  MRMs  enhanced  the  explorability  of  the  Afghanistan  environment  by  improving 
understanding. 

Understanding  was  measured  through  elicitation  and  analysis  of  individual  and  group  knowledge 
structures  using  the  Pathfinder  methodology  and  analysis  (Schvaneveldt,  1990).  Pathfinder  is  a 
knowledge  elicitation  technique  that  represents  domain  specific  knowledge  underlying  human 
performance.  Pathfinder  uses  pairwise  proximity  estimates  for  a  set  of  concepts  and  generates  a 
network  structure  where  the  concepts  are  nodes  and  the  relations  between  concepts  are  links  in 
the  network  structure.  Closely  related  concepts  are  represented  by  their  proximity  to  one  another 
in  the  Pathfinder  knowledge  structure. 

Pathfinder  has  been  in  use  for  more  than  20  years  to  represent  knowledge  structures  of  categories 
(Rubin,  1990),  scripts  (Durso  and  Coggins,  1990),  room  schemata  (Schvaneveldt,  1990),  and 
problem-solving  schemata  (Dayton  et  ah,  1990).  It  has  also  been  used  to  identify  novices  from 
experts  in  the  domains  of  air  combat  flight  maneuvers  (Schvaneveldt  et  ah,  1985),  computer 
programming  (Cooke  and  Schvaneveldt,  1988),  and  classroom  learning  (Goldsmith  and  Johnson, 
1990).  These  studies  have  indicated  that  Pathfinder  networks  represent  knowledge  structures  in  a 
meaningful  way;  it  identified  expert  and  novice  pilots  with  over  90%  accuracy  (Schvaneveldt 
et  ah,  1985)  and  accounted  for  55%  of  the  variance  in  students’  final  course  points  (Goldsmith 
and  Johnson,  1990). 

An  Afghanistan  expert  knowledge  structure  (figure  6)  from  a  former  special  adviser  to  the 
U.S.  embassy  in  Kabul  was  elicited  and  used  as  the  comparison  basis  for  all  subsequent  analyses. 
Improved  similarity  over  time  with  this  expert  referent  knowledge  structure  represents  improved 
understanding  and  learning. 
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Figure  6.  Expert  Afghanistan  environment  knowledge  structure. 

The  effect  of  using  COMPOEX  and  its  supporting  MRMs  was  examined  across  several 
groupings  that  occurred  during  LOE  2-3.  The  first  that  will  be  addressed  is  between  the 
leadership  and  planning  staff  groups.  The  leadership  group  consisted  of  the  three  leaders,  each 
directing  their  line  of  effort,  and  the  planning  staff  consisted  of  the  leaders’  supporting  staff.  The 
knowledge  structures  were  measured  on  the  Wednesday  and  Eriday  of  both  the  training  week  and 
the  execution  week.  An  analysis  of  similarity  with  the  expert  referent  or  the  correspondence  of 
links  between  the  leadership  and  staff  groups  with  the  expert  referent  knowledge  structures 
shows  a  general  positive  trend  for  both  the  leadership  and  staff  groups  over  time  (figure  7).  This 
increase  in  similarity  with  the  expert  referent  suggests  that  both  groups  are  improving  their 
understanding  of  the  Afghanistan  environment  as  they  interact  with  the  COMPOEX  system 
during  the  planning  process.  There  is  no  evidence  that  suggests  that  COMPOEX  is  affecting  the 
leadership  and  planning  staff  groups  differently. 
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Leader  &  Staff  Similarity  with  Expert  Referent 
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Figure  7.  Understanding  over  time  among  the  leadership  and  planning  staff 
groups. 

The  next  grouping  examined  is  between  the  participants  with  military  and  civilian/govemment 
background  and  experience.  There  were  15  participants  with  military  experience  and  11  with 
civilian/government  experience.  As  with  the  leadership  and  staff  results,  similar  patterns  emerge 
between  the  military  and  civilian  groups,  as  their  understanding  appears  to  have  similar  general 
positive  trends  (figure  8).  There  is  also  no  evidence  suggesting  that  COMPOEX  is  affecting  the 
military  and  civilian/govemment  personnel  any  differently  in  their  ability  to  improve 
understanding  over  time.  In  fact,  when  the  similarity  between  the  military  and  civilian  groups  is 
examined,  they  are  increasing  over  time  to  such  a  degree  that  there  is  a  statistically  significant 
positive  trend  {R  =  .90,  p  =  .05)  even  though  the  sample  over  time  is  quite  limited  (figure  9). 
This  suggests  that  understanding  is  improving  for  both  the  military  and  civilian  personnel  and 
shows  that  their  shared  understanding  is  also  improving  over  time. 
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Similiarity  Score 
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Figure  8.  Understanding  over  time  among  the  military  and  civilian/government  groups. 


Military  &  Civilian  Shared  Understanding  Over  Time 
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Figure  9.  Degree  of  similarity  between  knowledge  structures  of  the  military  and 

civilian/government  groups  over  time.  Trend  line  represents  a  significant 
positive  trend. 
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The  overall  understanding  results  across  all  participants  showed  a  significant  positive  trend  over 
time  (R^  =  .90,  p  =  .05)  (figure  10).  This  suggests  that  the  COMPOEX  system  had  a  positive 
influence  on  improving  understanding  among  the  users.  These  results  are  suggestive  because 
there  was  not  an  experimental  control  for  the  effects  of  participant  interaction  and  discourse.  The 
results  discussed  suggest  no  significant  differences  in  understanding  between  the  leadership  and 
planning  staff  or  the  military  and  civilian  groups.  Note  that  knowledge  structures  from  members 
of  the  comparative  planning  cell  were  collected  before  LOE  2-3.  The  comparative  planning  cell 
similarity  score  with  the  expert  referent  is  illustrated  in  figure  10,  and  is  similar  to  the  starting 
point  of  the  participants  in  EOE  2-3.  The  comparative  planning  cell  had  a  collective  22  years  of 
System  of  System  Analysis  experience  across  the  PMESII  areas,  and  had  been  engaged  in 
supporting  Afghanistan  related  missions  for  4  weeks.  In  the  9  days  between  the  first  and  last 
measure  for  the  participants  of  EOE  2-3,  they  nearly  doubled  their  similarity  score,  reflecting  an 
improved  understanding  of  the  Afghanistan  environment. 

Similarity  of  Overall  Understanding  with  Expert  Referent 
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Figure  10.  Understanding  over  time  across  all  participants.  The  trend  line  represents  a 
significant  positive  trend. 

Qualitative  changes  in  the  overall  knowledge  structures  are  evident  by  comparing  the  knowledge 
structure  at  T1  (day  2)  and  T4  (day  9)  (figure  11).  The  concepts  that  are  the  most  highly 
connected  concept  within  the  knowledge  structure  are  typically  related  to  their  level  of 
importance.  For  Tl,  it  is  the  “Security”  and  “Afghan  government”  concepts.  The  highest  degree 
centrality  concepts  at  T4  are  the  same  with  the  addition  of  the  “ISAF”  concept.  Further 
examination  of  the  knowledge  structures  reveals  that  even  though  some  of  the  same  key  concepts 
are  the  same  from  Tl  to  T4,  they  are  defined  differently.  Examination  of  the  direct  links  to  each 
of  these  concepts  shows  some  of  these  differences  for  the  “Afghan  government”  concept 
(table  1).  Similar  differences  are  evident  for  the  other  concepts  and  can  be  further  investigated  by 
comparing  the  knowledge  structures. 
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Figure  11.  Knowledge  structures  T1  (top)  and  T4  (bottom)  of  the  Afghan  environment  across  all  participants. 
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Table  1.  Concept  focus  on  the  “Afghan  government”  concept  for  the  Tl,  T4,  and  referent  knowledge  structures  in 
figures  1 1  and  6  (referent). 


Concept 

Tl  (Figure  11,  top) 

T4  (Figure  11,  bottom) 

Referent  (Figure  6) 

Afghan  government 

Development  of 
infrastructure 

— 

— 

Afghan  National  Security 
Forces 

Regional  power  brokers 

NATO  senior  civilian 
representatives 

Influence  operations 

— 

Drug  lords 

Afghan  population 

Governance 

_ ^ _ 

Note:  Check-marked  cell  represents  the  inclusion  of  that  concept  in  defining  the  “Afghan  governmenf’  concept. 

3.4  Trust 


As  previously  mentioned,  the  primary  purpose  of  the  MRMs  within  COMPOEX  is  to  facilitate 
the  development  of  explanations  and  help  enhance  understanding  of  the  planning  environment. 
This  enhanced  understanding  is  the  result  of  being  able  to  explore  the  operating  environment 
through  the  MRMs.  One  factor  that  may  influence  participants’  degree  of  understanding  is  trust 
in  the  MRMs.  Trust  may  mediate  the  extent  to  which  the  MRMs  can  be  used  to  explore,  and 
subsequently,  understand  the  planning  environment  (figure  12). 
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environment 

Trust  m  \  ^ 

Understand  the 

with  the  MRM 

^  the  MRM  ) 

environment 

Figure  12.  Theoretical  relationships  of  trust  as  a  mediating  variable  between  MRM  use  and 
increased  understanding  of  the  environment. 

Individuals  with  greater  trust  in  the  MRMs  may  gain  a  better  understanding  of  the  planning 
environment  because  they  believe  that  the  MRMs  generate  valid  and  reliable  outcomes.  As  such, 
they  are  able  to  form  an  understanding  of  the  interconnections  between  the  elements  in  the 
planning  environment.  Individuals  who  distrust  the  MRMs  may  not  see  these  interconnections 
because  they  feel  the  MRMs  do  not  accurately  represent  the  planning  environment. 

Trust  is  also  an  important  component  in  the  successful  use  of  the  MRMs.  If  trust  is  not  calibrated 
to  the  true  capability  of  the  MRMs,  operators  may  either  overrely  (misuse)  or  reject  (disuse)  the 
MRMs’  capabilities  (Lee  and  See,  2004).  These  usage  strategies  may  subsequently  affect 
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understanding.  Thus,  measuring  trust  may  allow  us  to  better  understand  how  participants  are 
using  the  MRMs  and  discern  whether  trust  is  a  mediating  factor  that  allows  participants  to 
understand  the  planning  environment.  The  following  section  describes  the  data  collection 
methods  and  results  surrounding  trust  in  the  MRMs. 

Participants  completed  a  13-item  human-machine  trust  measure  to  report  the  level  of  trust  they 
maintained  in  the  MRMs  during  the  exercise  (appendix  B).  This  was  a  modified  version  of  the 
Jian  et  al.  (2000)  original  human-machine  trust  scale.  Each  item  consisted  of  a  seven-point 
agreement  scale.  Five  of  the  items  were  negatively  framed  and  intended  to  measure  distrust  of 
automation.  For  instance,  one  of  these  items  asked  participants  to  rate  the  level  of  agreement 
with  the  following  statement:  “I  am  suspicious  of  the  MRMs,  action,  or  output.”  A  high  rating  on 
this  question  indicated  a  high  level  of  distrust.  The  remaining  eight  items  were  positively  framed 
and  were  intended  to  measure  trust.  A  high  rating  on  these  questions  indicated  that  the 
participants  had  a  high  level  of  trust  in  the  MRMs.  Ratings  for  the  negatively  worded  items  were 
reversed  scored  prior  to  calculating  total  trust  scores.  The  total  trust  score  was  calculated  by 
averaging  participants’  ratings  among  the  13  questions.  High  values  indicate  high  trust  in  the 
MRMs. 

Overall,  the  results  show  that  participants  maintained  moderate  amounts  of  trust  in  the  MRMs. 
Trust  ratings  clustered  around  the  midpoint  of  the  scale,  thus  indicating  that  participants  did  not 
overtrust  or  distrust  the  MRMs.  Further  analyses  revealed  that,  on  average,  senior  mentors 
exhibited  greater  trust  in  the  MRMs  than  planners  (figure  13). 
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Figure  13.  Trust  in  the  MRMs  as  a  function  of  participant  status. 
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Results  also  indicated  that  trust  varied  over  time  as  a  function  of  participant  status.  As  shown  in 
figure  14,  senior  mentors’  trust  in  the  MRMs  increased  slightly  over  time,  while  planners’  trust 
demonstrated  an  opposite  trend. 
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Figure  14.  Trust  in  the  MRMs  as  a  function  of  status  and  time. 

3.4.1  Predictive  Value 

Next,  the  predictive  value  of  the  MRMs  was  examined  to  determine  if  participants  believed  that 
the  MRMs  could  be  used  to  project  future  outcomes  in  the  planning  environment.  Predictive 
value  was  examined  using  a  single-item  indicator  (Question:  I  believe  that  the  MRMs  have 
predictive  value).  This  indicator  ranged  from  1  to  7.  Low  values  indicated  that  the  MRMs  did  not 
have  predictive  value,  whereas  high  values  indicated  that  the  MRMs  did  have  predictive  value. 
Figure  15  shows  how  predictive  value  varied  over  time  as  a  function  of  planning  experience. 
Note,  however,  that  civilian  and  military  planners’  perceptions  of  predictability  converged  at  the 
final  measurement  point.  Although  the  military  and  civilian  participants  were  not  significantly 
different  overall,  they  took  different  routes  over  the  event  regarding  their  perception  of  the 
MRMs  predictive  value  and  ended  at  the  same  level  at  the  end  of  the  event. 


18 


Predictive  Vaiue  as  a  Function  of  Pianning 
Experience  and  Time 
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Figure  15.  Predictive  value  as  a  function  of  planning  experience  and  time. 


3.4.2  Explorative  Value 

Next,  the  MRMs  were  examined  to  determine  if  they  allowed  participants  to  adequately  explore 
the  planning  environment  (Question:  I  believe  that  the  MRMs  enable  exploration  of  the 
environment.).  Explorability  was  also  measured  using  a  seven-point  agreement  scale.  Here,  high 
values  indicated  that  the  MRMS  allowed  for  adequate  exploration  of  the  planning  environment. 
We  hypothesized  that  there  would  be  a  positive  relationship  between  perceived  explorability  and 
trust.  However,  the  data  failed  to  support  this  hypothesis.  Though  there  was  a  positive  correlation 
between  trust  and  explorability,  it  failed  to  reach  significance.  Additional  analyses  revealed  that, 
on  average,  senior  mentors  had  higher  explorability  ratings  than  planners  (figure  16). 
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Explorative  Value  Across  Status 
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Figure  16.  Explorative  value  across  participant  status. 

3.4.3  Predictive  Value  Versus  Explorative  Value  and  Usage 

The  primary  purpose  of  the  MRMs  within  COMPOEX  is  to  facilitate  the  development  of 
explanations  and  help  enhance  understanding  of  the  operational  environment.  In  practice, 
however,  these  principles  do  not  always  hold  true.  Thus,  one  of  the  primary  questions  regarding 
COMPOEX  EOE  2-3  was  whether  participants  would  use  the  MRMs  for  exploratory  or 
predictive  purposes.  To  answer  this  question,  several  analyses  were  computed.  Eirst,  the 
perceptions  of  predictive  value  to  perceptions  of  explorative  value  were  examined.  As  shown  in 
figure  17,  participants  believed  that  the  MRMs  had  more  explorative  than  predictive  value 
(t  =  -1 1.9,  p  <  .01).  Even  though  the  explanatory  score  is  much  higher  than  the  prediction  score, 
the  prediction  score  is  right  at  the  neutral  score  of  4  and  is  not  significantly  lower.  This  suggests 
that  participants  are  still  using  the  MRMs  for  prediction  purposes  but  not  to  the  degree  they  are 
using  them  for  exploration. 
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Explorative  vs.  Predictive  Value 


2 

1  - 

T1  T2  T3  T4 

Time 

— Predict  — Explore 


Figure  17.  Comparison  of  explorative  value  vs.  predictive  value. 

Next,  we  assessed  participant’s  model  usage  strategy.  This  construct  was  measured  using  a 
single  item  indicator  that  ranged  from  1  to  7.  High  scores  indicated  that  the  MRMs  were  used  for 
explaining  the  environment,  whereas  low  scores  indicated  that  they  were  used  for  predictive 
purposes.  Ideally,  participants  should  have  used  the  MRMs  to  explain  the  planning  environment, 
rather  than  to  predict  future  states  of  the  environment.  As  shown  in  figure  18,  this  was  indeed 
true. 

Further  analyses  failed  to  show  significant  differences  in  model  usage  between  planning  types 
(military  versus  civilian)  or  levels  of  expertise  (senior  mentor  versus  planner).  This  lack  of 
significance  is  welcomed  and  suggests  that  participants,  regardless  of  status  and  experience,  used 
the  MRMs  to  develop  explanations  rather  than  for  prediction. 
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Model  Usage  Across  Time 
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Figure  18.  Model  usage  across  time. 

3.4.4  Workload 

An  examination  of  the  workload  across  several  dimensions  was  conducted  to  investigate  the 
effect  COMPOEX  had  on  workload  and  to  monitor  workload  across  the  duration  of  the  event. 
The  National  Aeronautics  and  Space  Administration  (NASA)  Task  Load  Index  (TLX)  (Hart  and 
Staveland,  1988)  workload  measure  was  used  and  administered  three  times  beginning  on  the 
Lriday  of  the  training  week  and  twice  in  the  execution  week  (Wednesday  and  Lriday).  The 
NASA-TLX  is  a  subjective  workload  assessment  measure  that  allows  users  to  perform  subjective 
workload  assessments  on  operator(s)  working  with  various  human-machine  systems.  A 
multidimensional  rating  procedure,  NASA-TLX  derives  an  overall  workload  score  based  on  a 
weighted  average  of  ratings  on  six  subscales.  It  can  be  completed  in  a  short  amount  of  time 
through  a  simple  computer  program. 

•  “Mental  demand”  refers  to  how  much  mental  and  perceptual  activity  was  required 
(thinking,  deciding,  calculating,  remembering,  looking,  searching,  etc.)  during  the  task.  The 
respondent  should  consider  whether  the  task  was  easy  or  demanding,  simple  or  complex, 
and  exacting  or  forgiving. 

•  “Physical  demand”  measures  the  required  physical  activity  in  relation  to  whether  the  task 
was  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  and  restful  or  laborious. 

•  The  amount  of  experienced  time  pressure  is  measured  by  the  “temporal  demand”  subscale. 
It  addresses  issues  such  as  whether  the  pace  of  interaction  was  slow  and  leisurely  or  rapid 
and  frantic. 

•  ’’Performance”  refers  to  how  successful  respondents  think  they  were  in  accomplishing  the 
goals  of  the  task  set  by  the  experimenter,  and  how  satisfied  they  were  with  their 
performance  in  accomplishing  these  goals. 
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•  The  criteria  of  “effort”  requests  the  respondents  to  assess  how  hard  they  had  to  work 
(mentally  and  physically)  to  accomplish  the  level  of  performance  they  achieved. 

•  The  “frustration”  level  measures  how  insecure,  discouraged,  irritated,  stressed  and  annoyed 
versus  secure,  confident,  relaxed,  and  complacent  subjects  felt  during  the  task. 

The  levels  of  total  workload  showed  a  gradual  increase  over  the  duration  of  the  event  along  with 
most  of  the  other  dimensions  of  workload  (figure  19).  These  results  are  primarily  attributed  to 
the  increased  tempo  of  the  event  as  the  planning  teams  neared  their  product  deadlines  at  the  end 
of  the  event.  Among  the  two  dimensions  showing  slight  but  nonsignificant  trends  in  the  opposite 
direction  (decreasing  over  time)  are  the  physical  demand  and  frustration  dimensions.  Although 
subtle  and  nonsignificant,  it  is  informative  that  frustration  appears  to  decrease  even  though  other 
types  of  workload  appear  to  be  on  the  increase.  This  suggests  that  some  degree  of  learning  and 
familiarity  were  gained  with  the  use  of  the  COMPOEX  tool  over  the  duration  of  the  event  even 
though  overall  workload  was  increasing  at  the  same  time. 

Workload  Dimensions  over  Time 


100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 


Figure  19.  Workload  results  by  dimension  over  time. 

Again,  these  results  are  only  suggestive  due  to  full  experimental  controls  not  being  in  place.  Any 
differences  among  the  workload  dimensions  were  examined  for  each  administration  between 
military  and  civilian  personnel.  There  were  only  two  significant  differences  between  those  two 
groups.  The  first  was  a  marginal  difference  during  the  middle  of  the  execution  week  (T3)  for  the 
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frustration  dimension.  Frustration  levels  for  the  military  experienced  participants  were 
marginally  higher  than  those  of  the  civilian/govemment  experienced  participants  (t  =  1.94, 
p  <  .07)  during  the  middle  of  the  execution  week  (Wednesday).  The  other  significant  difference 
was  evident  on  the  last  day  of  the  execution  week,  when  the  military  participants  reported  a 
significantly  higher  temporal  demand  than  the  civilian  participants  (t  =  3.17,  p  <  .01). 

3.5  User  and  Expert  Feedback 

The  past  three  results  sections  investigated  the  planning  products  produced  in  the  event,  how 
those  products  were  produced  through  different  patterns  of  interactions  with  others  and 
COMPOEX,  and  the  effect  those  interactions  had  on  key  human  performance  factors.  The  final 
result  section  will  examine  feedback  from  the  participants  on  COMPOEX  and  conclude  with  the 
results  from  the  expert  evaluators  that  observed  the  entire  event. 

3.5.1  User  Feedback 

On  the  last  day  of  the  execution  week,  participants  were  asked  15  questions  (appendix  C)  related 
to  the  capabilities  and  potential  benefits  of  using  COMPOEX.  Each  question  was  answered  on  a 
five-point  Eikert  scale  in  which  a  favorable  response  is  scored  higher  (e.g.,  5)  and  an  unfavorable 
response  is  scored  lower  (e.g.,  1)  (figure  20).  Eour  items  were  significantly  higher  than  a  neutral 
(3)  score,  Q2  (t  =  2.85,  p  <  .05),  Q3  (t  =  8.52,  p  <  .05),  Q4  (t  =  4.0,  p  <  .05),  and  Q5  (t  =  2.3, 
p  <  .05)  (table  2).  These  represent  capabilities  that  the  participants  felt  that  the  COMPOEX 
system  excelled  at  during  this  event.  On  the  other  hand,  there  were  two  items  that  were 
significantly  lower  than  a  neutral  (3)  score,  Q9  (t  =  -3.58,  p  <  .05)  and  Q15  (t  =  -3.3,  p  <  .05) 
(table  2).  These  represent  capabilities  that  the  participants  felt  that  COMPOEX  did  not  fully 
support.  Note  the  similarity  of  questions  14  and  15,  which  ask  about  the  appropriateness  level  of 
the  MRMs  for  operational  planning  (Q14)  and  strategic  planning  (Q15).  The  score  for 
operational  planning  was  not  significantly  below  neutral  but  was  lower  for  strategic  planning, 
suggesting  that  the  level  the  MRMs  represented  for  this  event  were  more  conducive  to 
operational  versus  strategic  planning. 
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Participant  Feedback  on  COMPOEX  Capabiiities 


5 

4.5 


4 

3.5 

2.5 
2 

1.5 
1 

0.5 


I  I  I  I  I  I  I 
I  I  I  I  I  I  I 


I  I  I  I  I  I  I 


Q1  Q2*  Q3*  Q4*  Q5*  Q6  Q7  Q8  Q9*  Q10  Q11  Q12  Q13  Q14  Q15* 


*  Significantly  different  from  the  score  of  3  at  the  p  <  .05  level. 

Figure  20.  Average  scores  for  each  COMPOEX  capability  item  answered  by  the  participants. 


Table  2.  Items  that  were  significantly  higher  and  lower  than  the  neutral  score  of  3  among  participants. 


High-Scoring  Items 

Low-Scoring  Items 

Q2.  COMPOEX  allowed  the  exploration  of  individual 
actions  alone  from  each  elements  of  power. 

Q9.  The  plan  contains  more  than  one  branch  to  account 
for  multiple  theories  and  ambiguity  in  some  data. 

Q3.  COMPOEX  allowed  the  exploration  of  multiple 
actions  together  from  a  single  element  of  power. 

Q15.  The  MRMs  are  at  an  appropriate  level  to  support 
strategic  level  planning. 

Q4.  COMPOEX  allowed  the  exploration  of  combined 
actions  from  different  elements  of  power. 

— 

Q5.  COMPOEX  allowed  the  exploration  of  modified 
strengths  and  sequence  of  actions  from  multiple 
elements  of  power. 

— 

There  were  several  differences  evident  among  these  items  between  military  and  civilian 
participants.  Among  the  15  questions,  the  military  participants  scored  significantly  higher  on 
four  of  the  questions  (table  3):  Q1  {t  =  1.73,  p  <  .05),  Q8  {t  =  1.73,  p  <  .05),  Q9  {t  =  1.74, 
p  <  .05),  and  Q15  (f  =  2.4,  p  <  .05).  The  civilian  participants  did  not  score  significantly  higher 
than  the  military  participants  on  any  of  the  items.  Two  of  these  items  are  the  same  as  the 
low-scoring  items  reported  in  table  2.  If  the  civilian  scores  are  removed  from  those  two  items, 
there  would  be  no  items  that  are  significantly  lower  than  a  neutral  score.  Therefore,  the 
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civilian/government-experienced  participants  felt  that  COMPOEX  did  not  fully  support  the 
capabilities  mentioned  in  Q9  and  Q15,  whereas  the  military-experienced  participants  felt  that  all 
items  were  supported  or  well  supported  by  COMPOEX. 

Table  3.  Items  that  were  significantly  higher  among  military-experienced  participants  vs.  civilian/government- 
experienced  participants. 


Ql.  COMPOEX  allowed  the  exploration  of  actions  from  the  full  range  of  DIME  elements  of  power  in  obtaining 
priority  effects. 

Q8.  Players  using  COMPOEX  created  the  most  likely  plan  to  achieve  the  priority  effects  based  on  COMPOEX 
derived  ToCs. 

Q9.  The  plan  contains  more  than  one  branch  to  account  for  multiple  theories  and  ambiguity  in  some  data. 

Q15.  The  MRMs  are  at  an  appropriate  level  to  support  strategic  level  planning. 


3.5.2  User  Satisfaction 

User  satisfaction  with  three  specific  tools  within  the  COMPOEX  system  was  measured  using  the 
System  Usability  Scale  (SUS)  (Brooke,  1996).  The  SUS  is  a  simple  10-item  scale  that  addresses 
subjective  assessments  of  usability  at  a  general  and  global  level  (appendix  D).  The  10-item  scale 
is  meant  to  be  used  as  a  single  score  and  not  broken  into  individual  item  scores.  The  three  tools 
that  were  assessed  were  the  primary  tools  used  in  EOE  2-3 — Sandbox,  CPT,  and  OET — with  its 
multiple  views.  The  OET  had  the  highest  usability  score,  60  out  of  100,  while  Sandbox  and  CPT 
both  had  scores  of  54.  All  the  user  satisfaction  scores  hovered  around  the  50%-60%  range, 
which  suggests  that  the  users  did  not  find  the  COMPOEX  interfaces  unusable,  did  not  find  them 
incredibly  user  friendly,  and  were  fairly  neutral  toward  the  COMPOEX  tool  interfaces.  This 
suggests  that  the  current  tool  interfaces  are  functional  but  have  room  for  improvement,  which 
may  further  improve  performance  parameters  such  as  ease  of  learning,  user  satisfaction, 
frustration,  trust,  and  understanding. 

Eurther  analyses  shows  that  there  was  a  significant  user  satisfaction  difference  between  civilian 
and  military  experienced  participants  for  the  CPT  (t  =  -2.0,  p  =  .05)  but  not  for  the  Sandbox  or 
the  OET  (figure  21).  These  results  indicate  that  the  participants  were  most  pleased  with  the 
interface  of  the  OET,  and  that  the  CPT  interface  created  different  user  experiences  between 
military  and  civilian  experienced  participants,  with  it  working  better  for  those  with  a  military 
background. 
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COMPOEX  User  Satisfaction 
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*  Significant  difference  between  civilian  and  military  groups  at  the  p  <  .05  level. 

Figure  21.  User  satisfaction  scores  between  civilian-  and  military-experienced 
participants  for  three  of  the  key  COMPOEX  tools  in  LOE  2-3. 


3.5.3  Expert  Evaluations 

Three  expert  evaluators  with  extensive  experience  in  planning  and  operational  support  observed 
both  the  training  and  execution  week.  Before  the  execution  week,  the  expert  evaluators 
completed  the  same  set  of  questions  that  the  participants  completed  in  appendix  C  on 
COMPOEX  capabilities  though  modified  to  address  traditional  planning  capabilities 
(appendix  E).  On  the  last  day  of  the  execution  week  and  after  observing  the  participants  using 
COMPOEX  for  the  duration  of  the  event,  the  expert  evaluators  completed  the  COMPOEX 
capabilities  questions  (appendix  C).  To  assess  any  differences  between  traditional  and 
COMPOEX-assisted  planning,  analysis  of  the  results  of  these  two  evaluations  were  completed. 
COMPOEX  was  evaluated  to  perform  items  Ql,  Q4,  Q5,  and  Q7  significantly  better  than 
traditional  planning  methods  and  tools  (figure  22).  Table  4  provides  the  content  of  the  items.  No 
traditional  planning  capabilities  were  evaluated  significantly  higher  than  COMPOEX  planning 
capabilities.  The  basic  trend  for  1 1  of  the  13  comparative  items  was  in  the  direction  favoring 
COMPOEX. 
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Expert  Evaluation  Ratings  between 
Traditional  &  COMPOEX  Planning 
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*  Significant  difference  between  COMPOEX  and  traditional  planning  at  the  p  <  .05  level. 

Figure  22.  Expert  evaluations  between  COMPOEX  and  traditional  planning  capabilities. 

Table  4.  Capabilities  that  were  evaluated  significantly  higher  for  COMPOEX-assisted  planning  compared  with 
traditional  planning. 

Ql.  COMPOEX  allowed  the  exploration  of  actions  from  the  full  range  of  DIME  elements  of  power  in  obtaining 
priority  effects. 

Q4.  COMPOEX  allowed  the  exploration  of  combined  actions  from  different  elements  of  power. 

Q5.  COMPOEX  allowed  the  exploration  of  modified  strengths  and  sequence  of  actions  from  multiple  elements  of 
power. 

Q7.  COMPOEX  allowed  the  development  of  a  comprehensive,  achievable  whole  of  the  government  plan  with 
multiple  lines  of  effort  and  sublines  of  effort. 


Comparing  these  results  with  the  results  from  the  participants  shows  agreement  on  items  Q4  and 
Q5,  which  focus  on  the  exploration  of  combined  actions,  strengths,  and  sequencing  of  actions 
from  multiple  elements  of  power.  The  other  two  capability  items  at  which  the  participants 
thought  COMPOEX  excelled  were  Q2  and  Q3  (table  2).  The  expert  evaluators  appear  to  agree  by 
evaluating  COMPOEX  considerably  higher  than  traditional  planning  on  these  but  did  not  reach 
statistical  significance,  primarily  due  to  the  small  sample  size  of  only  three  expert  evaluator 
ratings  for  this  analysis.  The  expert  evaluators  also  agreed  with  the  participants  on  the 
appropriateness  of  the  MRMs  at  the  operational  planning  level  (figure  22,  Q14)  but  less  so  at  the 
strategic  planning  level  (figure  22,  Q15). 
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4.  Conclusions 


The  COMPOEX  system  was  evaluated  in  LOE  2-3  on  what  products  it  facilitated,  how  it 
facilitated  those  products,  the  effect  it  had  on  understanding,  and  the  results  of  its  use  in  this 
2-week  planning  event.  The  analyses  across  each  of  these  areas  showed  that  COMPOEX 
consistently  had  positive  results  and  appears  to  support  the  planning  process  in  the  areas 
mentioned.  These  results  are  only  suggestive  because  full  experimental  controls  were  not  in 
place. 

The  output  and  products  that  COMPOEX  facilitated  were  evaluated  in  terms  of  the  time  and 
effort  to  develop  the  MRMs  and  the  planning  outputs  produced.  The  time  and  resources  required 
for  the  development  of  the  MRMs  compared  favorably  with  the  requirements  for  nodal  network 
development.  Hundreds  of  actions  were  examined  in  less  than  10  days  of  planning,  and  10 
effects  were  integrated  into  a  single  plan.  During  this  process,  a  fourth  line  of  effort  was 
identified  as  critical  to  the  operation  and  included  in  the  final  integrated  plan. 

The  coordination  that  was  facilitated  by  COMPOEX  showed  that  it  did  not  restrict  collaboration 
across  the  planners,  whether  it  was  across  lines  of  effort,  between  leadership  and  planners,  or 
between  military  and  civilian/govemment  participants.  Patterns  of  coordination  among  the 
participants  with  COMPOEX  also  showed  that  reliance  on  tool  support  personnel  declined  over 
time,  suggesting  that  participants  were  able  to  learn  and  increase  their  proficiency  with  the 
COMPOEX  tools. 

Interactions  with  COMPOEX  produced  several  positive  effects  among  the  participants  that 
support  the  planning  process.  Improved  understanding  across  both  civilian  and  military 
participants  as  well  as  across  experience  levels  was  evident  over  time.  Trust  in  the  MRMs  was  at 
moderate  levels,  and  no  difference  in  trust  was  evident  between  military  and  civilian  participants. 
As  intended,  participants  placed  greater  emphasis  on  the  explorative  use  of  the  MRMs  over  using 
them  for  prediction  purposes. 

The  use  of  COMPOEX  to  support  and  facilitate  planning  resulted  in  several  perceived  benefits 
and  strengths  by  the  participants  and  the  SMEs.  The  participants  evaluated  four  capabilities  at 
which  COMPOEX  significantly  excelled  and  two  at  which  it  showed  weaknesses.  Along  with 
these  capabilities,  moderate  levels  of  user  satisfaction  were  reported  with  the  user  interface  of  the 
COMPOEX  tools,  including  a  significant  difference  among  civilian  and  military  participants  for 
the  CPT.  Einally,  the  SMEs  rated  COMPOEX  significantly  better  on  four  capabilities  than 
traditional  planning  tools  and  procedures. 
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Appendix  A.  COMPOEX  Program  Components 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Sandbox 


Campaign  Planning  Tool 


Outcome  View 


Exploration  Analysis  View 


Graph  View 


1^1  “  ■AWwV'SV'iV. 


Panel  View 


r 


Model  Operation  View 


r 

.1 


•  Bs^lMtod  Power  Stnietures 
«Iraq  National  Power  Structure 


Legitimate  Economy 
Black  Economy 


Health  and  Sanitatien 


Appendix  B.  Trust  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Trust  Questionnaire 


Below  is  a  list  of  statements  for  evaluating  your  trust  in  the  in  the  COMPOEX  multi-resolution 
models  (MRM).  Please  rate  the  statements  on  the  extent  to  which  you  believe  they  describe  the 
MRMs.  You  may  refer  to  the  definitions  below  to  help  you  with  your  ratings. 

Definitions: 

Misleading  -  the  model  leads  one  to  commit  errors 
Unpredictable  -  the  system’s  future  behavior  is  unknown 
Inconsistent  -  the  model  output  is  erratic 

Valid  -  the  correct  actions  can  be  inferred  from  the  information  presented  by  the  model 
Dependable  -  the  system  is  worthy  of  user  trust 
Credible  -  the  system  is  worthy  user  confidence 

Reliable  -  the  information  presented  by  the  system  corresponds  to  reality 


Trustworthy  - 

-  the  system’s  behavior  is  reliable 

1. 

The  MRM  is  misleading. 

_| _ 1 _ 1 _ 1 _ 1 _ 

_1_  _1_ 

T  “  “  “  “ 

1  2  3  4  5 

not  at  all 

1  “ 

6  7 

very  much 

2. 

The  MRM  behaves  in  an  unpredictable  manner. 

_1_  _1_  _1_  _1_  _1_  _1_  _1_ 

1  1  1  1 

12  3  4 

not  at  all 

1  1  1 

5  6  7 

very  much 

3. 

I  am  suspicious  of  the  MRM’s  output. 

H  1  1  1 

-1-  -1-  -1- 

12  3  4 

not  at  all 

5  6  7 

very  much 

4. 

The  MRM’s  output  is  inconsistent. 

H  1  1  1 

-1-  -1-  -1- 

12  3  4 

not  at  all 

5  6  7 

very  much 

5. 

I  am  wary  of  the  MRM. 

H  1  1  1 

-1-  -1-  -1- 

12  3  4 

not  at  all 

5  6  7 

very  much 

6. 

I  am  confident  in  the  MRM. 

H  1  h - 1 

.  -1-  -1-  -1- 

12  3  4 

not  at  all 

5  6  7 

very  much 
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7.  The  MRM  provides  valid  models. 


+ - + - + - + - 

1  2  3  4  5 

not  at  all 

- + - + 

6  7 

very  much 

8.  The  MRM  is  dependable. 

_| _ 1 _ 1 _ 1 _ 1 _ 

_1_  _1_ 

T  “  “  “  “ 

1  2  3  4  5 

not  at  all 

1  “ 

6  7 

very  much 

9.  The  MRM  is  reliable. 

_| _ 1 _ 1 _ 1 _ 1 _ 

_1_  _1_ 

1  2  3  4  5 

not  at  all 

1  “ 

6  7 

very  much 

10.  I  can  trust  the  MRM. 

_| _ 1 _ 1 _ 1 _ 1 _ 

_1_  _1_ 

1  2  3  4  5 

not  at  all 

1  “ 

6  7 

very  much 

11.  The  MRM  has  credibility. 

_| _ 1 _ 1 _ 1 _ 1 _ 

_1_  _1_ 

1  2  3  4  5 

not  at  all 

1  “ 

6  7 

very  much 

12. 1  am  familiar  with  how  the  MRM  generates  its  models. 

_1_  _1_  _1_  _1_  _1_  _1_  _1_ 

1  1  1  1  1 

1  2  3  4  5 

not  at  all 

1  1 

6  7 

very  much 

13.1  believe  that  the  MRMs  have  predictive  value. 

_| _ 1 _ 1 _ 1 _ 1 _ 

_1_  _1_ 

1  2  3  4  5 

not  at  all 

1  “ 

6  7 

very  much 

14. 1  believe  that  the  MRMs  enable  exploration  of  the  environment. 

_1_  _1_  _1_  _1_  _1_  _1_  _1_ 

1  1  1  1  1 

1  2  3  4  5 

not  at  all 

1  1 

6  7 

very  much 

15.  Up  to  this  point  in  the  event,  I  have  used  the  MRM/s  primarily  for: 

_1_  _1_  _1_  _1_  _1_  _1_  _1_ 

1  1  1  1  1 

1  2  3  4  5 

Prediction 

1  1 

6  7 

Explanation 
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Appendix  C.  Conflict  Modeling,  Planning,  and  Outcomes  Experimentation 
Program  (COMPOEX)  Capability  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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COMPOEX  Capability  Questionnaire 


Please  provide  your  rating  from  the  drop-down  menu  for  each  item  below  that  best  describes  your 
observations  up  to  this  point  in  the  event. 

1.  COMPOEX  allowed  the  exploration  of  actions  from  the  full  range  of  diplomatic,  information, 
military,  and  economic  elements  of  power  in  obtaining  priority  effects. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

2.  COMPOEX  allowed  the  exploration  of  individual  actions  alone  from  each  elements  of  power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

3.  COMPOEX  allowed  the  exploration  of  multiple  actions  together  from  a  single  element  of  power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

4.  COMPOEX  allowed  the  exploration  of  combined  actions  from  different  elements  of  power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

5.  COMPOEX  allowed  the  exploration  of  modified  strengths  and  sequence  of  actions  from  multiple 
elements  of  power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 


6.  COMPOEX  allowed  the  integration  of  all  priority  effects  and  the  use  of  all  elements  of  power 
across  all  domains  of  the  environment. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

7.  COMPOEX  allowed  the  development  of  a  comprehensive,  achievable  whole  of  government  plan 
with  multiple  lines  of  effort  and  sub-lines  of  effort. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

8.  Players  using  COMPOEX  created  the  most  likely  plan  to  achieve  the  priority  effects  based  on 
COMPOEX  derived  Theory  of  Conflict(s). 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

9.  The  plan  contains  more  than  one  branch  to  account  for  multiple  theories  and  ambiguity  in  some 
data. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 
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1 .  Detailed  metrics  for  achieving  the  priority  effects  were  produced. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

2.  Key  assumptions  were  identified  and  challenged  during  the  planning  process  to  understand  the 
range  of  possible  outcomes. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

3.  COMPOEX  allowed  the  development  and  modification  of  feasible  Theories  of  the  Conflict. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

4.  These  Theories  of  Conflict  were  well  represented  in  a  Multi -resolution  model  (MRM). 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

14.  The  MRMs  are  at  an  appropriate  level  to  support  operation  level  planning. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 


15.  The  MRMs  are  at  an  appropriate  level  to  support  strategic  level  planning. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 


16.  Closing  comments  on  any  of  the  above  items  and/or  your  experience  with  the  COMPOEX 
system  during  this  event: 


Intentionally  left  blank. 
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Appendix  D.  System  Usability  Scale 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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System  Usability  Scale 


2. 1  found  the  system  unnecessarily 
complex 


3. 1  thought  the  system  was  easy 
to  use 


4. 1  think  that  I  would  need  the 
support  of  a  technical  person  to 
be  able  to  use  this  system 

5. 1  found  the  various  functions  in 
this  system  were  well  integrated 

6. 1  thought  there  was  too  much 
inconsistency  in  this  system 

7. 1  would  imagine  that  most  people 
would  learn  to  use  this  system 
very  quickly 

8. 1  found  the  system  very 
cumbersome  to  use 


9. 1  felt  very  confident  using  the 
system 

10. 1  needed  to  learn  a  lot  of 
things  before  I  could  get  going 
with  this  system 


Strongly 

agree 
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Appendix  E.  Traditional  Planning  Capability  Baseline  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Traditional  Planning  Capability  Baseline  Questionnaire 


Please  provide  your  rating  from  the  drop-down  menu  for  each  item  below  that  best  describes  your 
observations  of  traditional  planning  tools  and  processes. 

1.  Traditional  planning  allows  the  exploration  of  actions  from  the  full  range  of  diplomatic, 
information,  military,  and  economic  elements  of  power  in  obtaining  priority  effects. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 


2.  Traditional  planning  allows  the  exploration  of  individual  actions  alone  from  each  elements  of 
power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

3.  Traditional  planning  allows  the  exploration  of  multiple  actions  together  from  a  single  element  of 
power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

4.  Traditional  planning  allows  the  exploration  of  combined  actions  from  different  elements  of 
power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

5.  Traditional  planning  allows  the  exploration  of  modified  strengths  and  sequence  of  actions  from 
multiple  elements  of  power. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

6.  Traditional  planning  allows  the  integration  of  all  priority  effects  and  the  use  of  all  elements  of 
power  across  all  domains  of  the  environment. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

7.  Traditional  planning  allows  the  development  of  a  comprehensive,  achievable  whole  of 
government  plan  with  multiple  lines  of  effort  and  sub-lines  of  effort. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

8.  Players  using  traditional  planning  tools  and  processes  create  the  most  likely  plan  to  achieve  the 
priority  effects  based  on  COMPOEX  derived  Theory  of  Conflict(s). 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

9.  Plans  created  by  traditional  planning  tools  &  processes  contain  more  than  one  branch  to  account 
for  multiple  theories  and  ambiguity  in  some  data. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 
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10.  Detailed  metrics  for  achieving  the  priority  effects  are  produced  using  traditional  planning  tools 
and  processes. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 

1 1 .  Key  assumptions  are  identified  and  challenged  during  the  traditional  planning  process  to 
understand  the  range  of  possible  outcomes. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 


12.  Traditional  planning  allows  the  development  and  modification  of  feasible  Theories  of  the 
Conflict. 

1  2  3  4  5 

Strongly  Disagree  Strongly  Agree 


13.  These  Theories  of  Conflict  are  well  represented  in  traditional  plans. 

12  3  4 

Strongly  Disagree 


5 

Strongly  Agree 
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